<html>
<head>
  <title>METviewer Documentation</title>
  <link rel="stylesheet" type="text/css" href="mv_doc.css"/>
  <link rel="shortcut icon" href="include/ral_icon.ico" type="image/x-icon"/>
</head>
<body>
<p>

<p class="loc" style="padding-top:10px">
  <b>Location:</b> <a class="loc" href="index.html">Home</a> &#187; Database
  Loading Module
</p>
<hr/>

<h2>METviewer Documentation - Testing Module</h2>
<h3>Testing example - install and run on dakota</h3>
<p>
  Testing HOME directory: /d3/projects/METViewer/auto_test<br/>

  Testing data directory: /d3/projects/METViewer/test_data/load_data/load<br/>

  Branch to verify against: mv_2_5_dev
  <br/>
  Branch to verify : mv_2_6

</p>
<ol type="1">
  <li style="list-style-type: decimal;">
    check out and copy to the HOME script : auto_test.sh
  </li>
  <li style="list-style-type: decimal;">
    create a database for the branch to verify against (mv_2_5_dev):<br/>
    mysql -uUSER -p -e 'create database mv_test_mv_2_5_dev'
  </li>
  <li style="list-style-type: decimal;">
    clone and build the branch to verify against (mv_2_5_dev):<br/>
    ./auto_test.sh -UGIT_USER -t/d3/projects/METViewer/auto_test/METViewerTest -bmv_2_5_dev
    -Bmv_2_5_dev -l/d3/projects/METViewer/test_data/load_data/load -dmv_test_2_5_dev -uUSER
    -m/d3/projects/METViewer/auto_test/METViewer -pUSER -hHOST -P3306
  </li>

  <li style="list-style-type: decimal;">
    create a database for the branch to verify (mv_2_6):<br/>
    mysql -uUSER -p -e 'create database mv_test_mv_2_6'
  </li>

  <li style="list-style-type: decimal;">
    clone, build and verify the branch:<br/>
    ./auto_test.sh -UGIT_USER -t/d3/projects/METViewer/auto_test/METViewerTest -bmv_2_6 -Bmv_2_5_dev
    -l/d3/projects/METViewer/test_data/load_data/load -dmv_test_2_6 -uUSER
    -m/d3/projects/METViewer/auto_test/METViewer -pUSER -hHOST -P3306
  </li>
</ol>


<p>
  If the testing is done using crontab the gituser credentials should be stored in the git store
  - see https://git-scm.com/book/en/v2/Git-Tools-Credential-Storage. Also, auto_test.sh should be
  modified to send emails with testing results.
</p>


<h3>Testing - capture</h3>
<p>The testing modual is used to execute the regression testing of a specified
  version of METviewer. The capture tool is invoked by using the mv_test script.
  During test capture, system or processing errors will be reported but images and data are not
  verified.
  Verification of data or images is performed by the mv_compare script. The mv_test script
  essentially generates output files and
  copies them from the output directory to the appropriate directory test_cases directory.
</p>
<p>
  Capture and Verification are performed against a specified METviewer branch and tag. It is
  expected that the proper branch or tag has been
  previously fetched or cloned and checked out into the METviewer home directory that is specified
  with -m (the auto test script does check out specific code
  and then runs test capture and compare - see below). The METviewer test directory should also be a
  subdirectory that corresponds to the proper branch and
  tag, with an output directory that will contain the generated images, scripts and data for a given
  test run,
  along with the corresponding test_data (use case xml files) and load data (load specification xml
  files). The actual load data files are not
  yet source controlled and so they are linked from the path to the met data that is specified with
  the -l option. The auto_test script takes branch
  and tag parameters and creates the proper test subdirectory and reconciles properly with the
  remote repository. All the mv_test script needs is the
  proper test subdirectory. Successfully generated output images will be copied to the appropriate
  test_cases directory.
</p>
<h5>NOTE ABOUT GIT TAGS</h5>
<p>
  A short discussion of METviewer git tags is appropriate here. Git tags are not directly related to
  any git branches. They are different things. A git tag simply refers to a
  specific commit to the git repository. The branches that existed at the time of the tagging could
  all be deleted (except for the master) and the tag would still
  be a valid tag. For our purposes we name tags after our branches, i.e. MV_2_6_tagname or something
  like that. This creates an mnemonic
  relationship to a branch. Furthermore we only create tags after a commit to a specific branch that
  we then reference in the tag name. That tagged commit is
  a special point in the history of the repository, a point that we wanted to capture with a name on
  behalf of a branch. Specifying the tag and branch here
  allows us to set the repository pointer to the specific commit that we wanted to capture for a
  given branch. Including a branch name is only a mnemonic that
  reminds us which branch we were setting the tag on behalf of. The mnemonic allows us to organize
  our tests for regression checking against special times in
  the repository commit history.
</p>
<p>
  NOTE that since we want to do binary comparisons on captured test data the test data is saved in a
  directory structure that
  is mnemonically tied to the product branches or tags. The METviewer code under test is only ever
  checked out to one specific branch HEAD, or to a tag point, at a time.
  Its produced data and images may be compared, however, to data that was captured by a different
  version or tag point of METviewer code.

  This test directory structure might look like this, for example...
    <pre>
        METVIEWER_TEST_DIR/
            MV_2_5/
                HEAD/
                    test_data/
                        test_cases
                            diff_grouping
                                xml plot specification files
                                expected .png files
                            ens_ss
                                xml plot specification files
                                expected .png files
                            loading
                                xml plot specification files
                                expected .png files
                            phist
                                xml plot specification files
                                expected .png files
                            plot_afwa_thresh
                                xml plot specification files
                                expected .png files
                            rely
                                xml plot specification files
                                expected .png files
                            rhist
                                xml plot specification files
                                expected .png files
                            series_sum_stat
                                xml plot specification files
                                expected .png files
                    met_data
                        ens_vsdb
                            data dirs
                        noahmp
                            data dirs
                        precip_vsdb
                            data dirs
                        meso_vsdb
                            data dirs
                        grid_stat
                            data dirs
                        afwaoc
                            data dirs
                        point_stat
                            data dirs
                        ensemble_stat
                            data dirs
                    load_data
                        load
                            mv_mysql.sql
                            load_test.xml
                    output
                        data/
                            generated data files
                        plots/
                            generated .png files
                        scripts/
                            generated script files
                        sql/
                            generated sql files
                        xml/
                            generated xml files
                    R_work/
                            generated R scripts
                    R_tmpl   (link to METviewer R_tmpl)
                TAG1/
                    ....
                TAG2/
                    ....
            MV_2_6/
                ....
    </pre>

<h3>Testing - verify</h3>
<p>
  The mv_compare script accepts a branch and a tag that it uses to identify a test subdirectory and
  a second "expected" branch and
  tag that it uses to identify a comparison test directory. If tags are ommited the HEAD is used.
  The compare script looks for corresponding image files in the corresponding plots directories and
  does a binary comparison of
  the corresponding files. Differences will be reported as errors.
</p>
<h3>auto test</h3>
<p>
  The auto_test script defines the branch, optionally a tag, directories, and database credentials
  for a version under test
  and a comparison version. It performs the following steps...
<ul>
  <li>Remove any METviewer directory and METviewerTestSource directory</li>
  <li>Do a git clone into the METviewer home directory and checkout the corresponding branch or
    tag
  </li>
  <li>Do a git clone of the METviewer testing repository into the METviewerTestSource directory
    (this is not the test directory)
  </li>
  <li>If needed create a METVIEWER_TEST_DIR subdirectory for the specified branch and load common
    data from the METviewerTestSource into it
  </li>
  <li>Run the testing capture program to capture test data for the specified branch, and copy the
    data to the corresponding test subdirectory
  </li>
  <li>Run the testing verify program to verify the images in the two corresponding test
    subdirectories
  </li>
</ul>
<p>
  auto_test example: ./bin/auto_test.sh -Usomegituser -t/myhomedir/METViewerTest -bmv_2_5_dev
  -Bmv_2_5_dev -l/myhomedir/METViewerTestData -dmydb -umet_admin -m/myhomedir/METViewer -pdppassword
  -hdphost -P3306
</p>
<p>
  mv_test example:
  /bin/sh ./bin/mv_test.sh -t/myhomedir/METViewerTest/mv_2_5_dev/HEAD -m/myhomedir/METViewer -dmydb
  -umet_admin -pdppassword -hdphost -P3306 -l -c
</p>
<p>
  mv_compare example:
  /bin/sh ./bin/mv_compare.sh -m /myhomedir/METViewer -t /myhomedir/METViewerTest/mv_2_5_dev/HEAD -c
  /myhomedir/METViewerTest/mv_2_5_dev/HEAD
</p>
<p class="term">
  ---- Auto Test ----<br/><br/>
  Usage: auto_test.sh -U &#60git user&#62 -t&#60path to METviewer test directory&#62 -b&#60git
  branch&#62 -B&#60compare git branch&#62 -l&#60path to met data&#62 -d&#60mv_database&#62
  -m&#60path to METViewer home&#62 [-a address list] [-g&#60git tag&#62] [-G&#60compare git tag&#62]
  [-u&#60mv_user&#62] [-p&#60mv_passwd&#62] [-h&#60mv_host&#62] [-P&#60mv_port&#62] [-j&#60path to
  java executible&#62]<br/><br/>
  Where:<br/>
  -U &#60git user with access to github.com/dtcenter/METViewer.git and
  https://github.com/dtcenter/METViewer-test.git&#62 <br/>
  -t &#60path to METViewer test directory&#62 <br/>
  -b &#60git branch&#62 <br/>
  -B &#60compare git branch&#62 <br/>
  -l &#60path to met data&#62 causes the LoadDataTest submodule to be executed, gets met data
  from specified path <br/>
  -d &#60mv_database&#62 <br/>
  -m &#60path to METViewer home&#62<br/>
  [-a &#60address list&#62] commas separated email addresses - default sends output to console<br/>
  [-g &#60git tag&#62] default is HEAD<br/>
  [-G &#60compare git tag&#62] default is HEAD<br/>
  [-u &#60mv_user&#62] default is mvuser<br/>
  [-p &#60mv_passwd&#62] default is mvuser<br/>
  [-h &#60mv_host&#62] default is dakota.rap.ucar.edu<br/>
  [-P &#60mv_port&#62] default is 3306<br/>
  [-j &#60path to an alternate java executible&#62] default is found in PATH environment <br/>
  ---- Auto Test Done ----<br/>
</p>

<p class="term">
  ---- Capture ----<br/><br/>
  Usage: mv_test.sh -t&#60path to METViewer test directory&#62 -b&#60git branch&#62 [-g&#60git
  tag&#62] [-m&#60path to METViewer home&#62] [-d&#60mv_database&#62] [-u&#60mv_user&#62]
  [-p&#60mv_passwd&#62] [-h&#60mv_host&#62] [-P&#60mv_port&#62] [-j&#60path to java executible&#62]
  [-c] [-n] [-l]<br/><br/>
  Where:<br/>
  -t &#60path to METViewer test directory&#62 <br/>
  [-m &#60path to METViewer home&#62] default is /d3/projects/METViewer/src_dev/apps/METViewer<br/>
  [-d &#60mv_database&#62] default is mv_test<br/>
  [-u &#60mv_user&#62] default is mvuser<br/>
  [-p &#60mv_passwd&#62] default is mvuser<br/>
  [-h &#60mv_host&#62] default is dakota.rap.ucar.edu<br/>
  [-P &#60mv_port&#62] default is 3306<br/>
  [-j &#60path to an alternate java executible&#62] default is found in PATH environment <br/>
  [-n] causes no clean - i.e. test created data and plots will remain after the test completes,
  default is to clean prior to and after running<br/>
  [-l] causes the LoadDataTest submodule to be executed, default is to not load data, gets met data
  from METvviewer test directory meta_data<br/>
  [-c] Captures created output data and copies it into the test directory. Prior to doing this the
  compare will fail
  [-X] Do not compare output automatically - only create images
  <br/><br/>
  ---- capture Done ----<br/>
</p>

<p class="term">
  ---- Verify ----<br/><br/>
  Usage: mv_compare.sh [-b&#60git branch&#62] [-g&#60git tag&#62] [-B&#60compare git branch&#62]
  [-G&#60expected git tag&#62] [-m&#60path to METViewer home&#62] [-t&#60path to METViewer test
  directory&#62] [-j&#60path to java executible&#62]<br/>
  Where:<br/>
  -t &#60path to METViewer test directory&#62 <br/>
  -b &#60git branch&#62 <br/>
  -B &#60compare git branch&#62 <br/>
  [-g &#60git tag&#62] default is HEAD<br/>
  [-G &#60expected git tag&#62] default is HEAD<br/>
  [-m &#60path to METViewer home&#62] default is /d3/projects/METViewer/src_dev/apps/METViewer<br/>
  [-j &#60path to an alternate java executible&#62] default is found in PATH environment <br/>
  <br/><br/>
  ---- Verify Done ----<br/>
</p>

<p>
  The testing and compare modules produce output files (images, points, XML, data) and
  compares them with the expected output. In order for tests to pass, produced
  and expected files should be byte identical.<br/>
  <b style="color:#666666">NOTE:</b> R scripts create visually similar results but
  bitwise different image files on different platforms.<br/><br/>
  Testing submoduals:

<ul>
  <li><span class="code">LoadDataTest</span> recreates and refills mv_test
    database with MET output data and compares number of rows in each table
    with the expected number.<br/>
    &nbsp;&nbsp;Test data and XML configuration file are located in <span
        class="code">&lt;test_dir&gt;</span>/load_data directory
  </li>

  <li><span class="code">CreatePlotBatchTest</span> runs MVBatch with testing
    plot specification files and creates output files with the
    expected output. Any errors encountered with creating plots will be reported. Images are not
    compared<br/>
    &nbsp;&nbsp;plot specification files and expected output are located in
    <span class="code">&lt;test_dir&gt;</span>/plots_batch/&lt;test_type&gt;
    directory
  </li>

  <li><span class="code">ComparePlotBatchTest</span> \compares a test ROOT_DIR with a test
    COMPARE+DIR <br/>
    These directories are specified by the testdir and compare dir.
    &nbsp;&nbsp;plot specification files and expected output are located in
    <span class="code">&lt;test_dir&gt;</span>/test_data_test_cases/&lt;test_type&gt;
    directories.
  </li>

  <li><span class="code">TestMVServlet</span> simulates POST intermediate
    requests (ex. get list of variables), send them to MVServlet and
    compare servlet's response with the expected output.<br/>
    &nbsp;&nbsp;requests files and expected response are located in <span
        class="code">&lt;test_dir&gt;</span>/servlet/ directory
  </li>


  <li><span class="code">CreatePlotServletTest</span> simulates POST create a
    plot requests , send them to MVServlet and compare produces output
    files with the expected output.<br/>
    &nbsp;&nbsp;requests files and expected output files are located in
    <span class="code">&lt;test_dir&gt;</span>/plots_web/&lt;test_type&gt;
    directory
  </li>


</ul>
<p>Location of <span class="code">&lt;test_dir&gt;</span> : /d3/projects/METViewer/test_data/</p>

</body>
</html>
